Explatory Data Analysis for 2ForcedChoice Modal Experiments

Author

Utku and Sarah

1 Procedure for both experiments

  • Platform and environment
    • The experiments were conducted online in PCIbex (PennController for Ibex). The interface switched to fullscreen at the start of the main task and exited fullscreen at the end.
    • Participants were instructed to use a desktop or laptop with Google Chrome, a keyboard, and a mouse or trackpad in a distraction-free setting.
    • Total session time was approximately 25 minutes.
  • Consent and demographics
    • After an introduction screen, participants viewed and downloaded a consent form and indicated consent to proceed.
    • A brief demographics form collected age, gender, location (state and country), computer type, native language, and other languages.
  • Design and counterbalancing
    • Aim: the experiments tested the interpretive force of must compared to other modal elements, or the absence of a modal.
    • Participants were randomly assigned (between subjects) to one of four answer sets: bare, have to, will, prob (a separate CSV file per set).
    • Experimental, filler, and attention-check trials were fully intermixed and randomized for each participant.
    • On critical trials, the modal from a participant’s assigned set was contrasted with must (for example, will vs. must).
    • Filler trials featured the remaining modal options among the two responses.
    • Attention-check trials consisted of unambiguous deductive contexts to verify attention.
    • Counts: experimental items = X; fillers = X; attention checks = X.
    • Exclusion criterion: participants were excluded if they failed more than 20% of attention-check trials.
    • Each experimental item appeared in one of three context conditions: abductive, deductive, or inference.
    • Context conditions were assigned using a Latin square, so each participant saw exactly one context per item.
    • On every trial, the two response options were order-randomized.
    • Between-experiment differences:
      • Experiment 1 used SONA participants and included preambles that conveyed contextual information in an if-clause.
      • Experiment 2 used Prolific participants and included a shorter preamble consisting only of a well, then… phrase.
  • Instructions and practice
    • Participants read step-by-step instructions with two worked examples (one obvious and one less obvious).
    • Additional practice trials followed, using the same two-stage flow as the main task.
  • Trial structure (main task)
    • Context presentation: participants saw a short, three-turn conversation (Speaker A – Speaker B – Speaker A) that provided the relevant context.
    • Dialogue (reading) phase: participants pressed the space bar to advance or were auto-advanced after 120 s. A Dialogue RT (ms) was recorded from dialogue onset to advance.
    • Choice phase: two candidate sentence completions appeared simultaneously. Participants selected the option they judged most appropriate; a 60 s timeout applied. An Answer RT (ms) was recorded from option onset to response.
    • After the response, the screen cleared and the next trial began.
  • Breaks
    • Brief on-screen breaks were inserted at regular intervals (about every four trials). Participants resumed by pressing the space bar.
  • Measures and logging
    • For each trial, the following were logged: item and condition labels (Type, Group, Item, Inference, ConditionName, Condition), the three dialogue lines (D1–D3), both options (A1–A2), the selected answer, Dialogue RT, Answer RT, and TrialNumber.
    • Practice trials were logged with the same structure and marked as practice.
  • Debrief and redirect
    • Upon completion, results were submitted to the server, fullscreen ended, and participants were automatically redirected to a debrief and credit form hosted at UMD.

2 Experiment 1

2.1 Participant Accuracy in Check Items

must vs. prob
Total N = 14
Bin Count
[0.5,0.6) 3
[0.7,0.8) 6
[0.9,1] 5
must vs. will
Total N = 6
Bin Count
[0.7,0.8) 1
[0.9,1] 5
haveto vs. must
Total N = 9
Bin Count
[0.7,0.8) 2
[0.9,1] 7
bare vs. must
Total N = 11
Bin Count
[0.9,1] 11
Click to expand Demographics

Mean Age: 22.00 (18–49)

Subject Age Gender Location Computer Language Other language
S[51] 18 Male Maryland, United States of America Mac English Spanish (proficient), Darija (novice)
S[55] 30 Male MD, USA Mac English
S[60] 18 Unlabeled Maryland, United States PC (Lenovo) English Cantonese
S[5] 19 Female Maryland, USA PC English Hindi, Urdu
S[6] 18 Female MD, USA Windows English
S[9] 21 Female Maryland, USA Mac English
S[12] 49 MALE MARYLAND, USA LINUX PC ENGLISH SPANISH, ITALIAN, RUSSIAN NONE FLUENTLY
S[17] 19 Female Maryland, United States Windows Laptop English Spanish
S[21] 18 Male Maryland Laptop English
S[22] 18 Male Maryland, United States PC English
S[26] 18 Female Maryland, USA Lenovo English
S[27] 20 Female Maryland, USA Windows English Mandarin Chinese
S[31] 18 Female College Park, Maryland Mac English
S[34] 24 Female Maryland, United States Mac English Korean

Mean Age: 21.67 (18–36)

Subject Age Gender Location Computer Language Other language
S[53] 18 male maryland, usa pc english
S[57] 18 female Maryland, USA laptop english
S[59] 18 Female Maryland, USA Mac English Spanish
S[10] 21 Male Maryland, Prince George’s County PC English Spanish
S[11] 19 Male Maryland, US Windows English
S[20] 36 Male Baltimore, MD Mac English French, German

Mean Age: 19.00 (18–22)

Subject Age Gender Location Computer Language Other language
S[56] 18 female Maryland, USA lenovo thinkpad english chinese (beginner)
S[58] 19 Female MD, USA Mac English
S[1] 18 Female College Park, Maryland Mac English Cantonese
S[13] 19 Male College Park, MD, USA Mac English Gujarati
S[15] 20 Male College Park, MD Mac English Telugu
S[16] 22 Male MD, USA PC English German
S[28] 18 Female Maryland, USA Mac English Spanish
S[30] 19 Male Maryland, United States of America PC English
S[35] 18 Female College Park, MD Mac English Mandarin Chinese

Mean Age: 21.27 (18–40)

Subject Age Gender Location Computer Language Other language
S[52] 18 male MD, USA Thinkpad X1 Carbon English Mandarin, Spanish, Japanese, Hindi
S[54] 20 Female Maryland, United States PC English
S[3] 20 Male Massachusetts PC English
S[7] 18 woman MD, USA Windows English Japanese
S[8] 40 Female Maryland, USA PC English
S[14] 19 Female Maryland, United States Mac English Vietnamese
S[18] 26 male Maryland, USA PC English Spanish (L2 ~B2/C1)
S[19] 19 Female Maryland, USA PC English Spanish
S[24] 18 Female Maryland, United States Mac English
S[25] 18 female Maryland, USA Dell laptop English Vietnamese
S[32] 18 female MD, United States Mac English

2.2 Answer Choice Summary

2.3 Reading Times Summary

3 Experiment 2

3.1 Participant Accuracy in Check Items

must vs. prob
Total N = 21
Bin Count
[0.5,0.6) 1
[0.9,1] 20
must vs. will
Total N = 21
Bin Count
[0.7,0.8) 1
[0.9,1] 20
haveto vs. must
Total N = 21
Bin Count
[0.9,1] 21
bare vs. must
Total N = 21
Bin Count
[0.7,0.8) 3
[0.9,1] 18
Click to expand Demographics

Mean Age: 45.86 (27–73)

Subject Age Gender Location Computer Language Other language
S[1] 44 male TN, USA HP PC English
S[2] 49 male Michigan, USA PC English
S[3] 46 male texas, USA PC English
S[4] 52 male Tennessee, USA Mac English
S[5] 32 Male NC, USA PC English
S[6] 59 Female Alabama P.C. English
S[7] 73 Female Rhode Island PC English
S[8] 47 Male NY, USA PC English
S[9] 41 male MA, USA PC English
S[10] 48 Woman NY, USA PC English
S[11] 56 male Virginia, USA PC English
S[12] 50 female California, United States laptop English
S[13] 64 Male Illinois, United States PC English
S[14] 30 Female Hawaii, USA PC English
S[15] 34 Female New York, United States of America PC English
S[16] 40 Female New York, USA Laptop English
S[17] 42 Male Kentucky, USA Windows laptop English
S[18] 37 male Indiana, United states PC English
S[19] 27 woman US, TEXAS WINDOWS ENGLISH
S[20] 32 man Tennessee, United States PC English Spanish
S[21] 60 Male North Carolina, United States PC English

Mean Age: 46.10 (26–67)

Subject Age Gender Location Computer Language Other language
S[1] 31 female NJ USA Mac English
S[2] 52 Female Minnesota, USA PC English Beginning Spanish
S[3] 54 Male CA, USA PC English
S[4] 30 Female Kentucky, USA PC English
S[5] 43 female Connecticut, USA PC English
S[6] 26 male north carolina, united states pc english
S[7] 55 female Tennessee, USA Windows Laptop English
S[8] 37 Female Georgia, United States PC English
S[9] 36 female Maryland United states Chromebook English
S[10] 43 Female Texas, USA Mac English
S[11] 67 female Wisconsin, USA PC English Spanish
S[12] 46 male tn, usa PC English
S[13] 49 Female Massachusetts PC English
S[14] 56 female california united states pc english
S[15] 62 male ny, usa pc English
S[16] 51 Female Virginia Laptop English
S[17] 28 Female Maine, USA PC English
S[18] 59 male Tennessee, USA PC English
S[19] 52 female Virginia PC English English
S[20] 29 Female California United States Laptop English
S[21] 62 female Washington, USA PC English

Mean Age: 37.24 (22–71)

Subject Age Gender Location Computer Language Other language
S[1] 24 Male Florida, USA PC English
S[2] 45 Male Oregon, USA PC English
S[3] 29 f fl, usa pc english
S[4] 22 F CO, USA PC ENGLISH
S[5] 32 male Minnesota, USA PC English
S[6] 27 female Georgia, USA PC English
S[7] 38 Female Louisiana, United States PC English
S[8] 33 Woman Maryland, USA Mac English
S[9] 58 Male Georgia, USA PC English
S[10] 45 Male MS, USA MAC English
S[11] 53 Female United States Mac English English
S[12] 35 Non-Binary AFAB MN, USA Mac English
S[13] 42 woman Tennessee, USA PC(windows) English
S[14] 30 Female Indiana, USA PC English
S[15] 41 Female California, USA Mac English
S[16] 28 Male PA, USA Linux English
S[17] 47 Male Michigan, United States PC English
S[18] 71 Female Arizona, United States PC English
S[19] 26 Female Illinois, USA PC English Language
S[20] 25 Woman Delaware PC English
S[21] 31 Male MA, USA Windows English

Mean Age: 43.00 (21–66)

Subject Age Gender Location Computer Language Other language
S[1] 36 male CA, USA PC English
S[2] 35 Male AL, Geneva Chromebook English No others
S[3] 38 male Michigan, United States Of America PC English
S[4] 55 male California, USA PC English
S[5] 51 Male New York, United States PC English
S[6] 61 female California, US PC English
S[7] 21 male Georgia, United States PC English
S[8] 35 male VA, USA windows PC English
S[9] 45 Male Ohio, United States PC English
S[10] 66 Female Florida, United States PC English
S[11] 44 Male GA, USA PC English
S[12] 40 Female Louisiana, United States PC English
S[13] 25 Male United States PC English
S[14] 47 male New York, United States PC English
S[15] 46 Female Nevada, USA PC English
S[16] 47 Female Mississippi, United States PC English
S[17] 43 Woman SC, United States Windows English
S[18] 29 female ohio, USA PC english
S[19] 39 Male United States PC English
S[20] 45 Male Indiana, USA Laptop English
S[21] 55 female NY PC English English

3.2 Answer Choice Summary

3.3 Reading Times Summary

3.4 Age groups

4 Experiment 3

4.1 Participant Accuracy in Check Items

haveto vs. must
Total N = 21
Bin Count
[0.9,1] 21
bare vs. must
Total N = 21
Bin Count
[0.7,0.8) 1
[0.9,1] 20
Click to expand Demographics

Mean Age: 48.57 (35–73)

Subject Age Gender Location Computer Language Other language
S[1] 37 male California, USA PC english
S[2] 60 Female Idaho, USA PC English
S[3] 47 MALE BURKETVILLE, MD PC ENGLISH NONE
S[4] 38 male ma, usa mac english
S[5] 60 Female CA, USA Mac English
S[6] 50 Female California, USA Mac English
S[7] 42 female RI, USA PC English
S[8] 73 Female Kansas, USA laptop English
S[9] 35 male Colorado, United States PC English
S[10] 49 Male MN Mac English
S[11] 46 male Washington, USA pc English
S[12] 42 Female Virginia, USA PC English
S[13] 72 female Ohio, United States PC English
S[14] 44 female IL, USA PC English
S[15] 46 Male Michigan, United States PC English
S[16] 52 male CT USA pc English
S[17] 41 male Indiana. United states Laptop English
S[18] 58 female Michigan, United States PC English
S[19] 39 male NH, USA PC English
S[20] 44 female California, US 1pc English
S[21] 45 Male NC, USA PC English French (Intermediate), Spanish (Basic)

Mean Age: 48.52 (36–72)

Subject Age Gender Location Computer Language Other language
S[1] 49 woman NY, USA PC English
S[2] 43 Female New York, US HP English
S[3] 38 Female New York, United States PC English
S[4] 38 Male Arlington, USA PC English English
S[5] 71 Male Texas, United States PC English
S[6] 43 F MA, USA Mac English
S[7] 45 Female Maryland, USA Laptop%2C Windows English
S[8] 37 female La Porte, Texas PC English
S[9] 47 Male Florida, USA PC English
S[10] 52 female Missouri, USA Mac English
S[11] 51 male New Mexico, USA PC English
S[12] 44 Female NJ, United States PC English
S[13] 51 Male Nevada, USA PC English 0
S[14] 48 male OH, USA PC English
S[15] 36 female California, usa pc english
S[16] 72 Female Louisiana, United States PC English
S[17] 57 woman CA USA Mac English
S[18] 51 female michigan pc english
S[19] 60 male Missouri, USA PC American English Patios
S[20] 39 male RI, USA Pc English
S[21] 47 Male PA, USA MAC English

4.2 Answer Choice Summary

4.3 Reading Times Summary

5 Experiment 4

5.1 Participant Accuracy in Check Items

gotta vs. must
Total N = 21
Bin Count
[0.7,0.8) 2
[0.9,1] 19
gotta vs. must
Total N = 21
Bin Count
[0.7,0.8) 2
[0.9,1] 19
Click to expand Demographics

Mean Age: 46.00 (35–65)

Subject Age Gender Location Computer Language Other language
S[1] 65 female Arkansas, USA Windows English
S[2] 42 female MO, USA Mac English
S[3] 51 Female California, USA PC English
S[4] 62 male florida, usa pc english
S[5] 43 Female Kansas, USA PC English
S[6] 51 female IL, United States PC English
S[7] 35 Female OH, United States PC English
S[8] 50 male Texas, USA pc English Spanish
S[9] 39 female Willis, TX PC English
S[10] 39 Male Wyoming, USA PC English
S[11] 50 male New Jersey, USA PC English
S[12] 38 female Oklahoma, USA PC English
S[13] 37 Male IL, USA PC English
S[14] 37 Male New Jersey; United States Desktop PC English
S[15] 54 Female Illinois, United States Laptop English
S[16] 48 Male NH PC (windows) English
S[17] 40 Male Maryland, United States PC English
S[18] 43 Male Illinois, USA PC English
S[19] 50 male California, USA pc English Spanish
S[20] 37 Female North Carolina, USA PC English
S[21] 55 Female New York PC English

Mean Age: 50.05 (35–81)

Subject Age Gender Location Computer Language Other language
S[1] 41 female PA, USA PC English
S[2] 38 Male Illinois, USA PC English
S[3] 48 Male Wisconsin, USA PC English
S[4] 59 woman US PC English
S[5] 62 Female California, USA PC English
S[6] 43 male Florida, USA PC English
S[7] 47 Male Georgia, United States PC English
S[8] 51 male NC, Bladen PC English
S[9] 81 Female Tennessee, USA laptop English
S[10] 43 female Michigan, United States PC English
S[11] 44 Male Michigan, United States PC English
S[12] 38 male NC, USA PC English
S[13] 39 Male Kansas, United States PC English
S[14] 55 female Florida, USA PC English
S[15] 53 Woman Ohio, United States Mac English
S[16] 49 male florida, USA PC English
S[17] 53 Woman GA, United States PC Windows English
S[18] 47 Male Indiana, USA PC English
S[19] 65 female North Carolina, USA PC English
S[20] 60 Female MA, USA PC English
S[21] 35 female CA, US Mac English

5.2 Answer Choice Summary

5.3 Reading Times Summary